So...what is an Operator? 
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© Operators 

An Operator represents human operational 
knowledge in software, to reliably manage 
an application. 



LET'S GO BACK A FEW YEARS. 



CoreOS Blog 


j All CoreOS Posts 

Technical Posts 

Announcements 



«- Back to All Blogs 

Introducing Operators: Putting Operational Knowledge into Software 

November 03,2016 • By Brandon Philips 
Tags: announcements Operators 


A Site Reliability Engineer (SRE) is a person that operates an application by writing software. They are an engineer, a 
developer, who knows how to develop software specifically fora particular application domain. The resulting piece 
of software has an application's operational domain knowledge programmed into it. 

Our team has been busy in the Kubernetes community designing and implementing this concept to reliably create, 
configure, and manage complex application instances atop Kubernetes. 


o 


We call this new class of software Operators. An Operator is an application-specific controller that extends the 
Kubernetes API to create, configure, and manage instances of complex stateful applications on behalf of a 
Kubernetes user. It builds upon the basic Kubernetes resourcj^^^ntroller conceptsj^MfeLudes domain or 
application-specific knowledge to automate common tasks./ 


irca^^^ntroller concepts J^^^ud 

V © 


It builds upon the basic Kubernetes resource and controller concepts but includes domain or 
applicatiorvspecific knowledge to automate common tasks. 
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0 Domain or Application 
Specific Knowledge? 

Installing. 


Self-Heal. 


Scale (properly). clean Up. 
Update. 


Backup. 

Restore. 


etc. 
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An "Operator" takes 
advantage of what 
Kubernetes does best 


$ kubectl proxy 


$ curl localhost:8001 


jq .resources[].name 


$ curl http://localhost: 8001 /api/vl/ 
"bindings" 

"component statuses" 

"configmaps" 

"endpoints" 

"events" 

"limitranges" 

"namespaces" 

"namespaces/finalize" 
"namespaces/status" 

"nodes" 

"nodes/proxy" 

"nodes/status" 

"persistentvolumeclaims" 

"persistentvolumeclaims/status" 

"persistentvolumes" 

"persistentvolumes/status" 

"Pods" 




$ kubectl get pod kube-dns-1187388186-rrljb -n kube-system -o yaml 


(curl -XGET ../api/vl/namespaces/kube-system/pods/kube-dns-1187388186-rrljb) 


apiVersion: vl 
kind: Pod 
metadata: 

name: kube-dns-1187388186-rrljb 
namespace: kube-system 
ownerReferences:... 

Spec : 

Containers: 

name: kubedns 

image: gcr.io/google containers/k8s-dns-kube-dns-amd64:1.14.4 
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Operators take advantage of 

Custom Resource Definitions. 

...Formerly known as Third Party Resources (TPRs) 



CRDs allow us to extend the Kubernetes API 


Let's extend the Kubernetes API by creating our 
very own object/resource via CRDs. 


Create the CRD 


$ cat my-new-crd.yaml 

apiVersion: apiextensions.K8s.io/vlbetal 
kind: CustomResourceDefinition 
metadata: 

name: mysql.db.example.com 
Spec: 

group: db.example.com 
version: vl 
scope: Namespaced 
names: 

plural: mysqls 
singular: mysql 
kind: MySql 
short Names: 

- ms 


$ kubectl create -f my-new-crd.yaml 


Let's first verify the creation of the CRD object/resource. 


Verify CRD Creation via CLI 


$ kubectl get crd 


NAME 

mysql.db.example.com 


KIND 

CustomResourceDefinition.vlbetal.apiextensions.k8s.io 


Verify CRD Creation via API 


curl -XGET localhost:8001/apis/apiextensions.k8s.io/vlbetal/customresourcedefin it ions 


"kind": "CustomResourceDefinitionList" , 

"apiVersion": "apiextensions.k8s.io/vlbetal", 

"metadata": { 

"selfLink": "/apis/apiextensions.k8s.io/vlbetal/customresourcedefinitions 
"resourceVersion": "229273" 


"items": [ 

{ 

"metadata": { 

"name": "mysql.db.example.com", 

"selfLink": 

M /apis/apiextensions.k8s.io/vlbetal/customresourcedefinitions/mysql.db.examp1 
"uid": "8e4d17df-b085-lle7-9176-080027b424ef", 

"resourceVersion": "228836", 

"creationTimestamp": "2017-10-14T02:15:32Z" 

}, 
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Let's now actually verify our new mysql resource/object! 



Verify New Database Resource via 


$ kubectl get mysql 


No resources found. 


Verify New Database Resource via API 


curl -XGET localhost:8001/apis/db.example.com/v1/names paces/default/mysqls 


{ 

"apiVersion": "db.example.com/vl", 

"items": [], 

"kind": "MySqlList", 

"metadata": { 

"resourceVersion": "240591", 

"selfLink": "/apis/stable.example.com/vl/namespaces/default/mysqls" 

} 


Let's create a new database object. 


Create a new mysql object 


$ cat new-mysql-object.yaml 

apiVersion: "db.example.com/vl" 
Kind: MySql 
metadata: 

name: wordpress 
spec: 

user: wp 

password: secret 
too: bar 


$ kubectl create -f new-mysql-object.yaml 


Let's verify the creation of the mysql object. 


Verify database object via CLI 

$ kubectl get mysql 
NAME AGE 

wordpress 5s 

$ kubectl get mysql wordpress -o yaml 

apiVersion: db.example.com/vl 
Kind: MySql 
metadata: 

clusterName: "" 

creationTimestamp: 2017-10-14T03:23:26Z 

deletionGracePeriodSeconds: null 

deletionTimestamp: null 

name: wordpress 

namespace: default 

resourceVersion: "238701" 

selfLink: /apis/db.example.com/v1/namespaces/de fault/mysqls/wordpress 
uid: 0afdl584-b08f-lle7-9176-080027b424ef 
spec : 

foo: bar 

password: secret 
user: wp 


A Custom Resource needs a controller 

to ACT 

upon its presence. 


Kubernetes Controllers 



Current state of the cluster. 


Compare current state to desired state. 


Perform all the actions necessary to make 
current state meet desired state. 












We can understand controllers by 
studying the embedded controller 
"built-in" to Kubernetes: 


The ReplicaSet! 


What do ReplicaSets do? 


Redundancy 

Multiple running instances means failure can 
tolerated. 






Scale 


Multiple running instances mean more requests can 
handled. 






Sharding 


Multiple running instances can handle different parts 

of a computation in parallel. 












ReplicaSets in Action! 


kubectl create -f myfirstreplicaset.yaml 

kubectl scale replicaset myfirstreplicaset --replicas=3 



Secondary 
Resources 


Selector: app=myfirstap p 


ReplicaSetl 



apiVersion: extensions/vlbetal 
kind: ReplicaSet 
metadata: 

name: myfirstreplicaset 
spec: 

selector: 

matchLabels: 
app: mvfirstapp 
replicas 3 

template: 
metadata: 
labels: 

app: myfirstapp 
spec: 

containers: 

- name: nodejs 
image: myimage 


r 

\ 


\ 

r 

y 


Pod 


Pod 


Pod 

Label: app=myfirstapp 



Label: app=myfirstapp 


Label: app=myfirstapp 
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How do we accomplish this? 



O’REILLY 


PATTERNS FOR SCALABLE INFRASTRUCTURE AND APPLICATIONS 
IN A DYNAMIC ENVIRONMENT 


Justin Garrison & Kris Nova 


Chapter 4 

Designing Infrastructure Applications 


The reconciler pattern is a software pattern 
that can be used or expanded upon for 
managing cloud native infrastructure. The 
pattern enforces the idea of having two 
representations of the infrastructure—the first 
being the actual state of the infrastructure , 
and the second being the expected state of 
the infrastructure. 
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The reconciler pattern will force the engineer 
to have two independent avenues for getting 
either of these representations , as well as to 
implement a solution to reconcile the actual 
state into the expected state. 
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ReplicaSets in Action! 


Kube-API c.Watch(Replicaset) 


kubectl create -f myfirstreplicaset.yaml 


apiVersion: extensions/vlbetal 
kind: ReplicaSet 
metadata: 

name: myfirstreplicaset 
spec: 

selector: 

matchLabels: 
app: myfirstapp 
replicas: 3 
template: 
metadata: 
labels: 

app: myfirstapp 
spec: 

containers: 

- name: nodejs 
image: myimage 


Pod 


Label: 

app=myfirstapp 


Pod 


Label: app=myfirstapp 


0 < spec.replicas? 


1 < spec.replicas? 



2 < spec.replicas? 


Pod 


Label: 

app=myfirstapp 


> 


3 < spec.replicas? 


ReplicaSetController 





























































































ReplicaSets in Action! 


Kube-API c.Watch(Replicaset) 


ReplicaSetController 


kubectl create -f myfirstreplicaset.yaml 


apiVersion: extensions/vlbetal 
kind: ReplicaSet 
metadata: 

name: myfirstreplicaset 
spec: 

selector: 

matchLabels: 
app: myfirstapp 
replicas: 3 
template: 
metadata: 
labels: 

app: myfirstapp 
spec: 

containers: 

- name: nodej s 
image: myimage 


Pod 


Label: app=myfirstapp 


2 < spec.replicas? 



\3 s < spec .replica 


Label: app=myfirstapp 


Label: app=myfirstapp 


Pod 

> 





Pod 




























































Deployments! 









































Kube-API 



C\C\ 


:.Watch(Deployments) 


:.Watch(ReplicaSets) 


DeploymentController 



ReplicaSetController 
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10.3.0.1:443 


kubernetes-api 



Node2 


apiVersion: db . example.com/vl 
kind: MySql 
metadata : 

clusterName: "" 

creationTimestamp: 2017-10-14T03:47:21Z 

deletionGracePeriodSeconds: null 

deletionTimestamp: null 

name: wordpress 

namespace: default 

resourceVersion: "242282" 

selfLink: /apis/db.example.com/v1/names paces/de fault/mysqls/wordpress 
uid: 6228add3-b092-lle7-9176-080027b424ef 
spec : 

foo: bar 

password: secret 
user: wp 




10.3.1.100:443 



yourapp 



10.4.1.45.:443 



yourapp 


Pod 




Node3 


Pod 
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apiVersion: db.example.com/vl 
kind: MySql 
metadata: 

clusterName: "" 

creationTimestamp: 2017-10-14T03:47:21Z 
deletionGracePeriodSeconds : null 
deletionTimestamp: null 
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ACT? 

CREATE. 

READ. 

UPDATE. 


DELETE. 



But that's probably not enough.. 


Server startup/shutdown 
Mastering the mysqladmin administrative client 
Using the mysql interactive client 
User account maintenance 
Log file maintenance 
Database backup/copying 
Hardware tuning 
Multiple server setups 
Software updates and upgrades 
File system security 
Server security 
Repair and maintenance 
Crash recovery 
Preventive maintenance 
Understanding the mysqld server daemon 
Performance analysis 

Choosing what else to install (e.g. Apache, Perl +modules, PHP) 
Which version of MySQL (stable, developer, source, binary) 
Creating a user acccount for the mysql user and group 
Download and unpack a distribution 
Compile source code and install (or rpm) 

Initialize the data directory and grant tables with mysql_install_db 
Starting the server 
Installing Perl DBI support 
Installing PHP 
Installing Apache 

Obtaining and installing the samp_db sample database 


Securing a new MySQL installation 
Running mysqld as an unprivileged user 
Methods of starting the server 
Invoking mysqld directly 
Invoking safe_mysqld 
Invoking mysql.server 
Specifying startup options 
Checking tables at startup 
Shutting down the server 

Regaining control of the server if you can't connect 
Creating new users and granting privileges 
Determining who can connect from where 
Who should have what privileges? 
Administrator privileges 
Revoking privileges 
Removing users 

deciding/finding the Data Directory's location 
Structure of the Data Directory 
How mysqld provides access to data 
Running multiple servers on a single Data Directory 
Database representation 
Table representation (form, data and index files) 
OS constraints on DB and table names 
Data Directory structure and performance, resources, 
security 

MySQL status files (.pid, .err, .log, etc) 
Relocating Data Directory contents 


Creating new users and granting privileges 
Determining who can connect from where 
Who should have what privileges? 
Administrator privileges 
Revoking privileges 
Removing users 

Methods: mysqldump vs. direct copying 
Backup policies 
Scheduled cycles 
Update logging 

Consistent and comprehensible file-naming 
Backing up the backup files 
Off-site / off-system backups 
Backing up an entire database with mysqldump 
Compressed backup files 
Backing up individual tables 
Using mysqldump to transfer databases to another 
server 

mysqldump options (flush-logs, lock-tables, quick, 
opt) 

Direct copying methods 
Database replication (live and off-line copying) 
Recovering an entire database 
Recovering grant tables 
Recovering from mysqldump vs. tar/cpio files 
Using update logs to replay post-backup queries 
Editing update logs to avoid replaying erroneous 
queries 

Recovering individual tables 
Default parameters 
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To recap. 





Custom Resource Definitions (CRD) 

Third Party Resources (TPR) — 
















Custom Controller 















Your Knowledge! 




Operators! 



Why do Operators matter? 


We want an "as-a-service" platform experience! 


Build an ecosystem of software on Kubernetes that 
can be as easy, safe, and reliable to use and operate 

as a Cloud Service. 


Low-touch, remotely managed, one-click-updates. 


Operator Examples 


Q> SKbyCo^O 

© Prometheus 


EnaWed 1 nan«^ 
Show namespace 


Enabled 


Show namespace 



Vault 

0.1.3 by CoreOS, Inc 


Enabled 1 


Show namespace 




Super easy to deploy an Operator in a 

Kubernetes environment 



Create the CRD 


$ cat etcd-cluster-crd.yaml 

apiextensions.k8s.io/vlbetal 
kind: CustomResourceDefinition 
metadata: 
name: 

etcdclusters.etcd.database.coreos.com 
spec: 

group: etcd.database.coreos.com 
names: 

kind: EtcdCluster 
listKind: EtcdClusterList 
plural: etcdclusters 
shortNames: 

- etcdclus 

- etcd 

singular: etcdcluster 
scope: Namespaced 
version: vlbeta2 
versions: 

- name: vlbeta2 
served: true 
storage: true 



Deploy etcd Operator 

$ cat etcd-operator.yaml 

apiVersion: extensions/vlbetal 
kind: Deployment 
metadata: 

name: etcd-operator 
spec: 

replicas : 1 
template: 
metadata: 
labels: 

name: etcd-operator 
spec : 

containers : 

- name: etcd-operator 

image: quay.io/coreos/etcd-operator:v0.9.2 
command: 

- etcd-operator 

# Uncomment to act for resources in all namespaces. More information in doc/clusterwide.md 
#- -cluster-wide 
env: 

- name: MY_POD_NAMESPACE 
valueFrom: 

fieldRef : 

fieldPath: metadata.namespace 

- name: MY_POD_NAME 
valueFrom: 

fieldRef : 

fieldPath: metadata.name 



Deploy etcd Operator 

$ kubectl create -f etcd-operator.yaml 

$ kubectl get pods 


NAMESPACE 

default 


NAME 

etcd-operator-67666dc65f-xwfvq 


READY 

1/1 


STATUS 

Running 


RESTARTS 

0 
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View the etcdCluster 
Custom Resource 

$ cat etcd-instance.yaml 

apiVersion: "etcd.database.coreos.com/vlbeta2" 
kind: "EtcdCluster" 
metadata: 

name: "example-etcd-cluster" 
spec: 
size: 3 

version: "3.2.13" 



Deploy etcdCluster 


$ kubectl create -f etcd-instance.yaml 


$ 


kubectl 


NAMESPACE 

default 


get etcdcluster 


NAME 

myetcdcluster 


AGE 

Is 



How do you create your own Operator? 


RESOURCES 


type MyCustomResourceDefinition struct { 

// API obj kind & schema version 
metavl.TypeMeta 

// Standard object metadata (optional) 
Metadata api.ObjectMeta 
// Describe how the resource appears 
Spec 

vlbetal.CustomResourceDefinitionSpec 
// State of the CRD 
Status CustomResourceDefinitionStatus 

} 


CONTROLLERS 


for { 

current := getCurrentState() 
desired := getDesiredState() 
makeChanges(currentj desired) 
} 




Custom Operators require 
many building blocks and 
boilerplate code. 


...research/download tools to 
interact with the API. 


Officially-supported Kubernetes client libraries 


The following client libraries are officially maintained bv Kubernetes SIG API Machiner y. 


Language 

Client Library 

Sample Programs 

Go 

aithub.com/kubornetes/client-ao/ 

browse 

Python 

aithub.com/kubernetes-client/ovthon/ 

browse 

Java 

aithub.com/kubernetes-client/iava 

browse 

dotnet 

aithub.com/kubernetes-client/csharo 

browse 

JavaScript 

aithub.com/kubernetes-client/iavascriDt 

browse 

Community-maintained client libraries 

The following Kubernetes API client libraries are provided and maintained by their authors, not the Kubernetes team. 

Language 

Client Library 


Clojure 

aithub. com/vanatanl 6/cl i-kubernetes-a Di 


Go 

aithub.com/ericchiana/k8s 


Java (OSGi) 

bitbucket. ora/a mdatul abs/a md atu-kubernetes 


Java (Fabric8. OSGi) 

aithub.com/fabric8io/kubernetes-client 


Lisp 

aithub.com/brendandburns/cl-k8s 


Node.js (TypeScript) 

aithub. com/GoYQo/node-k8s-cl ient 




















Knowledge of informers/shared 
informers for object cache 
and event handling. 


Communicating desired state/actual 
state via annotations. 


Tracking kube-related resources. 


Test scaffolding & repo organization. 


CONFIDENTIAL Designator 


EXPLORER 

0 tpr.gc 

) X 


g $ m ••• 

-< OPEN EDITORS 


package tpr 


0 tpr.go pkg/operator/tpr 

2 






import ( 



* MEMHOG-OPERAT... ft afl 0 & 

4 

"fmt" 


A m app 

5 




8 memhog-operator.go 

6 

"k8s.io/client-go/kubemetes" 


7 

"k8s.io/client-go/pkg/api/errors" 


8 

"k8s.io/client-go/pkg/api/vl" 

tpr 

9 

"k8s.io/client-go/pkg/apis/extensions/vlbetal" 

app-monitor.yaml 

10 

) 



> m k8s 

11 




A pkg 

12 

func 

CreateTPR(clientSet kubernetes.Interface, name, version, desc string) (*vlbetal.ThirdPartyResource, error) { 


13 

// initialize third party resource if it does not exist 


14 

tpr. 

err 

:= clientSet.Extensions().ThirdPartyResources().Get(name) 

0 memhog-operator.go 

15 

if err l 

= nil { 

A tit operator 

16 



if errors.IsNotFound(err) { 

' ti> tpr 

17 




tpr := &vlbetal.ThirdPartyResource{ 


18 




ObjectMeta: vl.ObjectMeta{ 


19 




Name: name, 

0 app-monitor.go 

20 




>, 

0 controller.go 

21 




Versions: []vlbetal.APIVersion{ 

0 types.go 

22 




{Name: version}, 

> x utils 

23 




}, 


24 




Description: desc, 


25 




} 

D glide.lock 

26 





~ glide.yaml 

27 




result, err := clientSet.Extensions().ThirdPartyResources().Create(tpr) 

% LICENSE 

28 




if err != nil { 

Makefile 

29 




return nil, err 

0 memhog-operator.go 

31 




fmt.Printf ("CREATED: %#v\nFR0M: %#v\n", result, tpr) 

README.md 

32 



} else { 


33 




return nil, err 


34 



} 



35 

> else { 



36 


fmt.Printf ("SKIPPING: already exists %#v\n", tpr) 


37 

} 




38 





39 

41 

} 

return tpr, nil 
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We need an easier way 
to create Operators. 

We need an easier way 
to manage Operators. 



Operator Framework 

The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, 
automated, and scalable way. 

https://operatorhub.io/ Verified 


IQ Repositories 24 A People 59 @ Teams 2 


Pinned repositories 


5 operator-sdk 

SDK for building Kubernetes applications. 
Provides high level APIs, useful abstractions, and 
project scaffolding. 


IQ operator-lifecycle-manager 

A management framework for extending 
Kubernetes with Operators 


IQ operator-metering 

Operator metering is responsible for collecting 
metrics and other information about what's 
happening in a Kubernetes cluster, and providing 
a way to create reports on the collected data. 


• Go ★ 1.8k ^404 


• Go ★ 332 V141 


• Go *165 V 37 


Find a repository... 


Type: All- 


Language: All » 


community-operators 

The canonical source for Kubernetes Operators that appear on 
OperatorHub.io, OpenShift Container Platform and OKD. 

• Shell ★ 120 V149 Updated 6 hours ago 




Top languages 

• Go • Java • Shell t JavaScript 

• Python 


Most used topics Manage 

_kuhprnpfp^ 


operator-metering 


nnpratnr 





















Operator-SDK 



A 


ANSIBLE 

HELM 




Operator Lifecycle Manager (OLM) 

Enable cluster admins to manage 
Operators on any Kubernetes cluster 
(dependency management). 


Operator Metering 

Record historical cluster usage and 
generate usage reports showing usage 
breakdowns by pod or namespace over 
arbitrary time periods. 


Explore Operators 


OperatorHub.io is a home for the 
Kubernetes community to share 
Operators. 

Find an existing Operator or list your 
own today. 


OperatorHub.io 


Q. Search OperatorHub... 


Contribute 


Welcome to OperatorHub.io 

OperatorHub.io is a new home for the Kubernetes community to share Operators. Find an 
existing Operator or list your own today. 


CATEGORIES 

52 ITEMS 


VIEW SS v- SORT A-Z v 

Al/Machine Learning 




Application Runtime 




Big Data 

o 

% 

aws 

Cloud Provider 


Database 

Aqua Security Operator 

AtlasMap Operator 

AWS Service Operator 

Developer Tools 

provided by Aqua Security, Inc. 

provided by AtlasMap 

provided by Amazon Web 
Services, Inc. 

Integration & Delivery 

The Aqua Security Operator 

AtlasMap is a data mapping 


Loggings Tracing 

runs within Kubernetes cluster 

solution with an interactive 

The AWS Service Operator 

Monitoring 

and provides a means to 

web based user interfac 

allows you to manage AV 

Networking 

OpenShift Optional 

Security 

Storage 

9 

T 

fi 

Streaming & Messaging 

Camel K Operator 

CockroachDB 

Community Jaeger Operator 
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Check out learn.openshift.com/operatorframework 


Operator SDK with Go 


Step 1 of 7 

Creating a New Project 

Let's begin by creating a new project called myproject: 


oc new-project myproject ✓ 


Let's now create a new directory in our $GOPATH/src/ directory: 


mkdir -p $GOPATH/src/github.com/redhat/ ✓ 


Navigate to the directory: 


cd $GOPATH/src/github.com/redhat/ ✓ 


Create a new Go-based Operator SDK project for the PodSet: 


operator-sdk new podset-operator —type=go / 


Navigate to the project root: 


cd podset-operator 


CONTINUE 


Terminal Console 3 Ef + 

go: finding golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7 
go: finding github.com/dgrijalva/jwt-go v3.2.0+incompatible 

go: finding github.com/prometheus/client_model V0.0.0-20190812154241-I4fe0dlb01d4 

go: finding k8s.io/utils v0.0.0-20191010214722-8d271d903fe4 

go: finding github.com/coreos/prometheus-operator V0.34.0 

go: finding github.com/imdario/mergo V0.3.8 

go: finding k8s.io/kube-state-metrics vl.7.2 

go: finding github.com/prometheus/common V0.7.0 

go: finding gopkg.in/yaml.v2 v2.2.4 

go: finding go.uber.org/atomic vl.4.0 

go: finding github.com/json-iterator/go vl.1.7 

go: finding go.uber.org/multierr vl.1.0 

go: finding github.com/modern-go/reflect2 vl.0.1 

go: finding gomodules.xyz/jsonpatch/v2 v2.0.1 

go: finding github.com/google/uuid vl.1.1 

go: finding github.com/pkg/errors V0.8.1 

go: finding golang.org/x/text V0.3.2 

go: finding k8s.io/kube-openapi v0.0.0-20190918143330-0270cf2flcld 
go: finding github.com/cespare/xxhash/v2 v2.1.0 
go: finding github.com/beorn7/perks vl.0.1 
go: finding gopkg.in/fsnotify.vl vl.4.7 

go: finding github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7efldd 

go: finding github.com/hashicorp/golang-lru V0.5.3 

go: finding github.com/matttproud/golang_protobuf_extensions vl.0.1 

go: finding github.com/prometheus/procfs V0.0.5 

go: finding github.com/google/go-cmp V0.3.1 

go: finding golang.org/x/sys V0.0.0-20191028164358-I95ce5e7f934 

go: finding github.com/go-openapi/spec V0.19.4 

go: finding github.com/emicklei/go-restful v2.11.1+incompatible 

go: finding github.com/go-openapi/jsonpointer V0.19.3 

go: finding github.com/go-openapi/jsonreference V0.19.3 

go: finding github.com/go-openapi/swag V0.19.5 

go: finding github.com/PuerkitoBio/purell vl.1.1 

go: finding github.com/mailru/easyjson V0.7.0 

go: finding github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 

□ 
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